test: the ordered oracle, and three run-shape defects that green a whole session (#432) - #906
Conversation
998439f to
eb84040
Compare
|
Rebased onto the rebased #905, which is now on Counts in the body above predate that and are low. From disk: The four extra are #903's twin arriving through Re-gated:
|
jdatcmd
left a comment
jdatcmd
left a comment
There was a problem hiding this comment.
One blocker, and it makes the harness unusable for the way people actually run it: pytest -k fails a healthy run. Everything below I reproduced by running it against eb84040.
BLOCKER: deselection is not loss, and the run-shape guard cannot tell them apart
$ pytest -q test_layer.py # control
16 passed in 0.56s
$ pytest -q test_layer.py -k "refus"
.
VACUITY: 15 collected test(s) never reported an outcome, so the run lost them
silently: test_layer.py::test_a_counted_assertion_passes, ...
1 passed, 15 deselected in 0.04s
exit=1
A green tree, a normal invocation, exit 1, and a message asserting the run "lost them silently" when pytest has just printed 15 deselected on the line above. _RunShape records the collected node-ids before deselection is applied and reconciles against reported outcomes, so every deselecting flag trips it: -k, -m, --deselect, --lf, --ff.
That is worse than a false red. It is a false red from the guard whose subject is false greens, and the first thing anyone will do is learn to unset the plugin or stop using -k. --lf is how you iterate on a failure; -k is how you run one test. The README documents neither as unsupported.
-x does not trip it on an all-green run — nothing stops early — but -x with a failure deselects the remainder and will stack this on top of the real failure it was meant to surface.
The fix is to subtract what pytest reports as deselected. pytest_deselected(items) gives it to you directly, and there is a pytest_collection_modifyitems hook that runs after deselection.
MAJOR: the sorted() guard catches exactly one spelling
The claim is that sorted() feeding ordered_rows is uncollectable. Run against the real layer:
expect.ordered_rows(sorted(got), sorted(want), ...) -> refused ✓
g = sorted(got); expect.ordered_rows(g, w, ...) -> PASSES <- same defect
got.sort(); expect.ordered_rows(got, want, ...) -> PASSES <- same defect
Binding to a local defeats it, and list.sort() in place defeats it. Both are the more natural way to write the thing the guard exists to stop, because a reader who has already been told not to sort inline will move the sort up a line.
A guard that catches one spelling of a defect reports coverage for the class. VACUITY_MODES.md lists this as closed; it is closed against the inline form only, and the document should say which.
While checking, the caught case reports itself as 1 collected test(s) never reported an outcome, so the run lost them silently rather than naming the sorted() violation. The refusal is right and the diagnosis points at the wrong thing — the same failure mode as the false red above, and probably the same code path.
MAJOR: the stack states two different refused-mode counts
README "refused by this layer today" VACUITY_MODES §2
pr905 23 23 of the 79 consistent
pr906 23 27 of the 79 <- disagree
#906 moves §2 from 23 to 27 and leaves README.md:7 at 23. Both are counts of the same quantity, in two files, in one stack — which is the failure TESTS.md's totals line produced three times today and which your own #908 proposes to fix by deriving the number instead of writing it. It is worth noticing that it happened here, in the document about claims that cannot be checked.
The -n 4 question I could not answer, said as unknown rather than guessed
The README documents pytest -n 4. I did not test the run-shape guards under xdist, so I do not know whether they reconcile correctly there, silently do not run, or trip the same false red per worker. Given that the deselection bug exists in the serial path, the xdist path is worth a measurement before this lands — and it is your measurement to take, since you have the crashed-worker fixture already.
What is right
The two-oracles-are-different-instruments test is the correct shape and the argument behind it — two oracles that always agree are one oracle with two names — is the sort of premise most suites never state. Feeding a permutation and requiring the set oracle to accept while the sequence oracle rejects is exactly right.
Finding that a line regex for sorted( fired inside its own makepyfile string, and moving to an AST walk because of it, is the right response to the right problem. My finding above is that the AST walk did not go far enough, not that it was the wrong move.
Asks
- Subtract deselected items before reconciling. Blocking.
- Catch the bound-local and
.sort()shapes, or narrow the claim inVACUITY_MODES.mdto the inline form. - Reconcile
README.md:7with §2, and consider deriving it per #908. - Say what happens under
-n 4, with a measurement. - #905's three asks land underneath this one; the dead
nodes == 0block is still present here at:418.
Both harnesses, same change, per the house rule.
|
Your findings all stand and I am fixing them. But I ran a mutation census over the whole layer before you posted, and it turned up something that sits underneath your finding #1 and is already on The censusEvery The seven are Each is pinned by a test named after it, and each test passes with the guard removed. Why — and it is not subsumptionI assumed subsumption and was wrong. With the left-sentinel guard neutered, the inner run's output still contains: pytest prints the enclosing function's source in a traceback, including lines that never executed. result.stdout.fnmatch_lines([f"*{p}*" for p in patterns])so the pattern matches the neutered guard's own string literal, in the traceback. Every message in That is the defect the helper exists to prevent, in the helper. Your The fix, verified both waysAnchor to pytest's error-line prefix so the phrase must appear in a raised message: result.stdout.fnmatch_lines([f"E*{p}*" for p in patterns])Clean discrimination on all four. Scope, and it is not this PR
Three of my own instruments failed getting to thisSaid because the census is only worth what the refuter is worth:
Your findingsAll reproduce and all are going in the fix. Two I want to name:
And you refuted a claim while checking it: I will hold both PRs until the whole set is fixed and re-censused, rather than pushing fixes one at a time into a review you have already done once. |
… the order scan Answers the review on commandprompt#906. The first of these is the one that mattered most, and the framing in the review was better than mine. the run-shape guard failed any run that used -k ----------------------------------------------- `pytest_collection_modifyitems` fires BEFORE pytest's own -k and -m filtering has removed anything, so every deselected test looked like a test that vanished without reporting: $ pytest -q test_layer.py -k "refus" 1 passed, 15 deselected VACUITY: 15 collected test(s) never reported an outcome ... exit 1 A false red on a healthy run, produced by the guard whose whole subject is false greens. It is worse than a missed true red, because the response to it is to stop using -k, and then to stop using the plugin. `pytest_deselected` is where pytest distinguishes the two, so the guard now learns the difference there. Asking for a subset is a deliberate act by whoever typed the command; a test lost to a crashed worker is not. $ pytest -q -k "refus" 46 passed, 62 deselected exit 0 Three arms, and the third is the one that matters: subtracting the deselected ids is only correct if a genuinely lost test is still caught, so one arm deselects AND kills a worker in the same run and requires the red. Without it an over-broad subtraction would pass every other arm and quietly retire the guard. the order-killer scan saw one spelling of three ------------------------------------------------ It caught `expect.ordered_rows(sorted(got), ...)` and walked past both of these, which read as more careful code than the version it did catch: g = sorted(got) # bound to a name first expect.ordered_rows(g, want) got.sort() # killed in place; the call site is unchanged The scan now tracks names bound to an order-killing call and names sorted in place, within one function body. Its limits are named in VACUITY_MODES.md 2.1 and pinned by a test, so "one function deep" cannot quietly become a claim of completeness. It compares line numbers, so a name sorted AFTER the claim is not refused: a guard against false greens has no business emitting a false red. False-positive budget over the real corpus before trusting it: 0 hits in 12 files. the inventory could count one mode in two states ------------------------------------------------- Rebasing onto commandprompt#905 put the new counting rule over a document where modes had actually moved, and it reported 25 refused and 50 unrefused out of 72 named. The three extra are section 3's back-references -- "`X` is now closed" -- which point at modes that moved into section 2. Section 1a now says section 2 wins, and section 3's total is the ids it names minus the ids section 2 claims. 25 + 47 = 72, and both harnesses implement it. This only became visible because the totals were gated; the same document previously carried 27 and 24 in different places with nothing to catch either. Proved able to fail, each mutation asserted applied by md5 and restored byte-exact: pytest_deselected stops subtracting 2 arms redden (both -k spellings) the killed-name scan forgets its names 2 arms redden (bound and in-place) harness_selftest 387 passed + 0 failed + 0 unrunnable, rc=0 pytest corpus 108 passed serial, under -n 4, and with --pgc-expect-tests pytest -k 46 passed, 62 deselected, exit 0 docs_style 9 checks PASSED shellcheck -S error clean Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uf6UoeBRZYLQZa4KxNiw8a
eb84040 to
c9d396c
Compare
|
Fixed at
Three arms. The third is the one I would look at: subtracting the deselected ids The order-killer scan saw one spelling of three. Both of the ones you named Limits are named in VACUITY_MODES.md 2.1 and pinned by a test, so "one function One the rebase turned up. Putting #905's counting rule over a document where That only surfaced because the totals were gated. The same document previously |
jdatcmd
left a comment
jdatcmd
left a comment
There was a problem hiding this comment.
The blocker is fixed and the order-killer scan is widened. Verified at c9d396c2 by running both, with exit codes measured rather than piped.
The blocker: pytest -k no longer fails a healthy run
pytest -q test_layer.py 16 passed exit 0
pytest -q test_layer.py -k refus 1 passed, 15 deselected exit 0 <- was exit 1
pytest -q test_layer.py -k "not zzz" exit 0
pytest -q test_layer.py --deselect ... exit 0
My own instrument note: my first pass at this read $? after a pipeline into tail and reported exit=0 for a run that had actually exited 4. Exit codes are now measured with the command's own status. A pipeline's $? is the last stage's, and that would have let me approve a blocker as fixed on the strength of tail succeeding.
And the guard is not merely quieter — it still fires. --pgc-expect-tests 999 against a 16-test file exits 4 with ERROR: collected 16 test(s) but expected 999, and --pgc-expect-tests 16 exits 0. Subtracting deselected items did not cost the guard its teeth, which is the half that mattered: the easy fix would have been to stop reconciling at all.
The order-killer scan now catches all three spellings
expect.ordered_rows(sorted(got), sorted(want), ...) refused
g = sorted(got); expect.ordered_rows(g, w, ...) refused <- was invisible
got.sort(); expect.ordered_rows(got, want, ...) refused <- was invisible
Both evasions I found are closed, and test_runshape.py is 9/9, so the run-shape arms still hold.
One minor thing, not blocking
When the session aborts at collection, the lost-tests reconciliation fires as well:
ERROR: collected 16 test(s) but expected 999. ...
VACUITY: 16 collected test(s) never reported an outcome, so the run lost them silently: ...
The first line is right and comes first, so nobody is misled about the cause. The second is technically true and diagnostically wrong — those tests did not run because the session refused to start, not because the run lost them. It is the same shape as the deselection bug one level along: reconciling against a collected set that was reduced for a legitimate reason. Suppressing the reconciliation when the session aborted before running anything would cost a line.
Verified at this head
harness_selftest 387 passed + 0 failed + 0 unrunnable
pytest corpus 108 passed
CI 12/12 SUCCESS, CLEAN
Merge order, because this one bites
#905 first, then this. When #905 merges, do not use --delete-branch: deleting the base of a stacked PR auto-closes the child irreversibly. Retarget this to main first, or merge the chain with merge commits rather than rebase-merges.
Approving. The sorted() line-regex firing inside its own makepyfile string, and your moving to an AST walk because of it, was the right response to the right problem — my finding was only that the walk stopped one level short, and it does not any more.
|
#909 is merged as All four open PRs conflict on the same one line. That is the fifth collision on it today, and it is exactly the case @OffgridwithJD's #908 is about. Do not pick a side of the conflict. Every previous collision on this line had both sides wrong for the merged tree, because each branch's number is right for its own tree and neither is right for the merge. Recount from the corpus with the gate's own function: python3 -c "
import sys, pathlib; sys.path.insert(0, \"test/pytest\")
import test_docs_cover_the_corpus as g
c = g.corpus_tests(pathlib.Path(\"test/pytest\"))
print(f\"files={len(c)} tests={sum(len(v) for v in c.values())}\")"
The corpus gate will tell you if you get it wrong, in both harnesses. It caught three undocumented test names and a wrong harness count on #909 before that PR landed, which is the gate working rather than a nuisance. Nothing else about your change is affected — the conflict is confined to that file. Ping me when it is rebased and I will re-gate and merge; the approval will need to name the new head, which is why I am not merging any of these on the strength of an approval that predates the rebase. |
…ommandprompt#432) An enumeration ran in the audit container against pytest 9.1.1, xdist 3.8.0 and psycopg 3.3.5, with every mode required to be DEMONSTRATED BY AN ACTUAL RUN rather than described. It produced 79 modes, 73 of them executed, and a refusal design for 74. WHAT DID NOT RUN, SAID FIRST. The adversarial stage that would have attacked each refusal was cut off by a session limit: 148 attacks started, 0 completed. So the summary line reading "defeated: 0" counts zero defeats out of ZERO ATTEMPTS, and none of the 74 designs has met an adversary. VACUITY_MODES.md says so in its own section rather than leaving the number to be misread. CHECKING THE LAYER AGAINST THE INVENTORY FOUND THREE GAPS. Two are here; the third landed first, in commandprompt#897, and this commit now defers to it. expect.num(-1, -1) passed. cursor.rowcount is -1 when no count is available and 1 for an unfetched SELECT, and both are numbers. expect.rowcount now refuses the sentinel and says what it is. A broad except was forbidden IN A COMMENT, which enforces nothing. After any failed statement psycopg raises for every later one, so one `except Exception` hides the real error and all its successors. It is now uncollectable. plan_marker(absent=True) returned a pass against []. That gap is closed on main by commandprompt#897's own guard, so the two tests this commit wrote for it are DROPPED rather than shipped beside it -- two tests for one property under two names is what makes a corpus hard to read, and the doc gate would then require documenting both. Independent discovery is worth recording; a duplicate test is not. AND THE BROAD-EXCEPT GUARD IMMEDIATELY REJECTED CODE ALREADY ON MAIN. Rebasing it onto 6364e22 turned the whole run red at collection: ERROR: ... test_build_refusal.py:340 except Exception catches Exception broadly -- catch the specific exception class instead. That is my own arm from commandprompt#897, catching a failed make_cluster broadly. Narrowed to (FileNotFoundError, RuntimeError, OSError) -- a missing pg_config raises FileNotFoundError out of the subprocess layer, measured, and anything else now escapes and fails loudly, which is what should happen to an error the arm did not predict. The guard earned its place before this branch was opened. Written first as a line regex, the guard fired on the forbidden shape appearing inside a pytester.makepyfile STRING and so rejected the layer's own tests. It now parses with ast, where a handler inside a string literal is not an ExceptHandler node. A line regex over source cannot tell code from a string, which is the same mistake as matching a plan by substring. One assumption of mine was refuted by checking: expect.rows does NOT sort, so ordered claims are testable through it. The collapse comes from CALLERS sorting, which test_native_projection.py does deliberately. Verified: harness_selftest 342 passed + 0 failed + 0 unrunnable PASSED docs_style 9 checks PASSED pytest 76 passed serial, 76 passed -n 4, marker cleared for each shellcheck -S error -s bash test/*.sh test/selftest/*.sh exit 0 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uf6UoeBRZYLQZa4KxNiw8a
Answers the review on commandprompt#905. Four defects, three of them in the layer's own guards and one in the document that describes them. expect.refusal matched the whole traceback, not the error --------------------------------------------------------- `expect.refusal` built its patterns as `*{p}*` and ran them over the pytest output. pytest prints the ENCLOSING FUNCTION'S SOURCE in a traceback, so a pattern naming the thing the guard refuses matched the fixture's own source line and passed whether or not the guard fired. Anchored to `E*{p}*`, which is the error line pytest actually emits. This is on main from commandprompt#897 and it is the largest of the four: 13 merged arms were asserting nothing. Verified by neutering each guard in turn -- with the guard live the arm passes, with it removed the arm now fails. Four arms tested, all four held. plan_marker carried a dead branch --------------------------------- `nodes == 0` could not be reached: `if not nodes:` above it returns first. Removed. `nodes == 0` now occurs zero times and `if not nodes:` once. the broad-except scan missed every tuple handler ------------------------------------------------- `except (ValueError, Exception):` is as broad as `except Exception:` and the scan walked past it, because it inspected the handler type only when that type was a bare Name. It now inspects each member of a Tuple. All five spellings verified: `Exception`, `(ValueError, Exception)`, `BaseException` and the bare `except:` are refused; `except ValueError:` still passes as the control. the inventory could not be checked, so it drifted -------------------------------------------------- README.md said 23 refused modes and VACUITY_MODES.md said 27, and a reader could check NEITHER, because the document offered no rule for what counts as a mode. That is the defect this directory exists to refuse, committed by the document describing the refusal. Section 1a now states the rule -- a mode is a backticked kebab-case identifier of three or more words -- and reconciles the totals against it: 21 refused, 51 not, 72 named, against 79 the enumeration produced. The seven never written down are named as a gap rather than counted as coverage. The numbers are now gated in both harnesses, because a total nobody recomputes goes stale the same way twice: * `test/pytest/test_docs_cover_the_corpus.py` -- four arms over the table, the README, the gap arithmetic, and the prose totals outside the table. * `test/selftest/350-the-pytest-corpus-must-be.sh` -- the same rules, and this is the copy with teeth: nothing in the gate runs pytest. The two implementations disagreed, and the disagreement was the point. The bash reader took the first number on the line and returned 2 and 3 for totals of 21 and 51 -- the digits inside "named in section 2". Its fixture could not see it because there the label digit and the value were both 2, so there is now an arm whose only job is to tell those two readings apart. Proved able to fail, each mutation asserted applied by md5 and restored byte-exact: 1a states 22 refused, disk has 21 arm reddens a refused mode id loses its backticks arm reddens README drifts back to 23 arm reddens the gap row closes on its own arm reddens section 2's opening drifts back to 23 arm reddens the closing paragraph drifts back to 23 arm reddens TESTS.md drifts back to 23 arm reddens harness_selftest 387 passed + 0 failed + 0 unrunnable, rc=0 pytest corpus 84 passed serial and under -n 4 docs_style 9 checks PASSED shellcheck -S error clean Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uf6UoeBRZYLQZa4KxNiw8a
The largest gap the mode inventory found: lib.sh has had pgc_seq_hash, diff_query_ordered and pgc_check_ordered_oracle since commandprompt#418, and the port had no way to express an ordering claim at all. A test naming ORDER BY and comparing sorted lists cannot fail on order. FOUR MECHANISMS, each with a red test that fails without it. expect.ordered_rows compares two sequences IN ORDER and names the position of the first difference rather than saying two hashes differ. It refuses two empty sequences, and it refuses a sequence whose elements are all identical: such a sequence reads the same forwards and backwards, so an ordering claim about it cannot fail. That second refusal is pgc_check_ordered_oracle's premise applied to a caller's data instead of to a fixed fixture. expect.ordering_observable is that premise itself, ported: read the same rows both ways and require the two to differ before relying on order. expect.row_set is the counterpart, and the reason it exists is that ignoring order should be DECLARED. pgc_check_ordered_oracle asserts three things and this is the third: the set oracle must be order-blind BY DESIGN. Without a control proving the two instruments differ, an ordered oracle could quietly be implemented as a set one and every ordering test in the tree would go silent while staying green. The arm that pins it runs both oracles over the same forward and reverse sequences and requires one to ignore the reversal and the other to catch it. And a collection-time ast scan refuses sorted(), set() or frozenset() feeding ordered_rows. The helper is order-sensitive, so the collapse is introduced at the call site: ordered_rows(sorted(got), sorted(want)) reads like an ordering claim and is not one. Parsed rather than grepped, for the reason the broad-except scan is. test_native_projection.py now declares which oracle it wants. Its fetch helper returns rows in query order and the assertions use row_set, because the bash original uses pgc_set_hash and is order-blind. Sorting inside a fetch helper is exactly how an ordering claim silently becomes a set one, so the helper no longer does it. The property comparison against the bash suite still reports every property covered. 38 tests, green serially and under four xdist workers. VACUITY_MODES.md records the mode as closed and the count as 24 of 79. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uf6UoeBRZYLQZa4KxNiw8a
The layer refused vacuous tests. It did not refuse vacuous runs. These three shapes each turn an entire session green rather than one test, which is why VACUITY_MODES.md ranked them above the rest of the backlog combined. Reconcile reported node-ids against collected ones. Measured on a 6-test corpus under -n 2 --max-worker-restart=0: bare pytest exits 1 and names the crash, but reports 5 node-ids for 6 collected and never mentions test_d, which was assigned to the dead worker and never ran. The loss is silent, not the crash. A suite whose crash lands on an already-failing test therefore reports what you expected while running fewer tests than you wrote. Fail a parametrize over an empty set, with its own message. A corpus glob matching nothing gives one s and exit 0, and the cause a reader needs is the corpus, not the marker. Fail a skip arriving during fixture setup. A session fixture calling pytest.skip() skips every dependent test, so "the cluster would not start" becomes exit 0. expect.cannot_run stays available and records a counted assertion instead of skipping. Two measurements shaped the implementation. Under xdist the workers collect, not the controller, so the guard was present and blind until it listened to pytest_xdist_node_collection_finished. And the state lives on a per-config plugin instance: pytester.runpytest() runs the inner session in-process, so module-level sets leaked between these tests and the sessions they drive, and the corpus reported 44 passed while exiting 1. 44 passed, exit 0, serially and under -n 4, with exit codes read without a pipe. VACUITY_MODES.md now records 27 of 79 refused. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uf6UoeBRZYLQZa4KxNiw8a
These live here rather than in the base branch's copy of test_guards_pinned.py
because the guards they pin do not exist until the ordered oracle does.
Both are unreachable by SUBSUMPTION rather than untested, which is why the
outcome alone cannot pin them. Neuter ordered_rows' both-empty guard and the
UNOBSERVABLE guard fires on the same input, because every element of an empty
sequence is trivially the same. Neuter ordering_observable's and the
forward == reverse AssertionError fires, because two empty readings are equal.
Either way the inner run still fails, so an arm asserting only failed=1 still
passes and the guard is pinned by nothing.
A mutation census over the full stack found exactly these two still unheld once
the rest of the layer was pinned, which is how they were found rather than by
reading.
full stack, census before 17 guards 5 HELD 12 UNHELD
full stack, census after 18 guards 18 HELD 0 UNHELD
71 passed, exit 0, serial and under -n 4.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uf6UoeBRZYLQZa4KxNiw8a
… the order scan Answers the review on commandprompt#906. The first of these is the one that mattered most, and the framing in the review was better than mine. the run-shape guard failed any run that used -k ----------------------------------------------- `pytest_collection_modifyitems` fires BEFORE pytest's own -k and -m filtering has removed anything, so every deselected test looked like a test that vanished without reporting: $ pytest -q test_layer.py -k "refus" 1 passed, 15 deselected VACUITY: 15 collected test(s) never reported an outcome ... exit 1 A false red on a healthy run, produced by the guard whose whole subject is false greens. It is worse than a missed true red, because the response to it is to stop using -k, and then to stop using the plugin. `pytest_deselected` is where pytest distinguishes the two, so the guard now learns the difference there. Asking for a subset is a deliberate act by whoever typed the command; a test lost to a crashed worker is not. $ pytest -q -k "refus" 46 passed, 62 deselected exit 0 Three arms, and the third is the one that matters: subtracting the deselected ids is only correct if a genuinely lost test is still caught, so one arm deselects AND kills a worker in the same run and requires the red. Without it an over-broad subtraction would pass every other arm and quietly retire the guard. the order-killer scan saw one spelling of three ------------------------------------------------ It caught `expect.ordered_rows(sorted(got), ...)` and walked past both of these, which read as more careful code than the version it did catch: g = sorted(got) # bound to a name first expect.ordered_rows(g, want) got.sort() # killed in place; the call site is unchanged The scan now tracks names bound to an order-killing call and names sorted in place, within one function body. Its limits are named in VACUITY_MODES.md 2.1 and pinned by a test, so "one function deep" cannot quietly become a claim of completeness. It compares line numbers, so a name sorted AFTER the claim is not refused: a guard against false greens has no business emitting a false red. False-positive budget over the real corpus before trusting it: 0 hits in 12 files. the inventory could count one mode in two states ------------------------------------------------- Rebasing onto commandprompt#905 put the new counting rule over a document where modes had actually moved, and it reported 25 refused and 50 unrefused out of 72 named. The three extra are section 3's back-references -- "`X` is now closed" -- which point at modes that moved into section 2. Section 1a now says section 2 wins, and section 3's total is the ids it names minus the ids section 2 claims. 25 + 47 = 72, and both harnesses implement it. This only became visible because the totals were gated; the same document previously carried 27 and 24 in different places with nothing to catch either. Proved able to fail, each mutation asserted applied by md5 and restored byte-exact: pytest_deselected stops subtracting 2 arms redden (both -k spellings) the killed-name scan forgets its names 2 arms redden (bound and in-place) harness_selftest 387 passed + 0 failed + 0 unrunnable, rc=0 pytest corpus 108 passed serial, under -n 4, and with --pgc-expect-tests pytest -k 46 passed, 62 deselected, exit 0 docs_style 9 checks PASSED shellcheck -S error clean Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uf6UoeBRZYLQZa4KxNiw8a
c9d396c to
f563a60
Compare
jdatcmd
left a comment
jdatcmd
left a comment
There was a problem hiding this comment.
Re-approving at f563a60b. The blocker fix, the widened sorted() scan and the run-shape guard I verified at c9d396c2 all stand; the rebase moved only the totals line, and it is recounted rather than picked:
stated : 120 tests in 8 files
counted: 120 tests in 8 files | harness=105 product=15 | inputs == sum(buckets)
gate : stated == on disk, undocumented = none
That you needed the recount twice — 113/8 at one commit and 120/8 at the next, neither authorable in either branch — is the sixth collision on that line today and settles #908's case without further argument.
git rebase --onto <new-base> <old-base> for the stacked child is worth putting somewhere durable. A plain git rebase <new905head> replays the child's own already-rebased commits against themselves, which is a conflict that looks like a content problem and is not.
Holding the merge until the checks land.
2 participants
The ordered oracle the port did not have, and three run-shape defects that green a whole session rather than one test.
Stacked on #905 (the vacuity inventory), because the run-shape guards came out of that inventory's ranking. Review #905 first; this adds 17 tests on top of it.
1.
lib.shhas two oracles and this port had onepgc_set_hashsorts before hashing, so a bash test namingORDER BYand comparing withdiff_querycannot fail on order —pgc_seq_hashanddiff_query_orderedare the ones that can.test_ordered.pyports that pair and its premise check, nine tests.The property worth reading twice is
test_the_two_oracles_are_different_instruments: two oracles that always agree are one oracle with two names, and a suite built on them would pass every ordering claim by construction. The test feeds both a permutation and requires the set oracle to accept while the sequence oracle rejects.test_layer_refuses_sorting_the_input_to_an_ordered_claimmakessorted()feedingordered_rowsuncollectable, found by AST — becauseordered_rows(sorted(got), sorted(want))reads as an ordering claim and is not one.And the AST scan matters for the same reason the broad-
exceptscan in #905 does. A line regex forsorted(fired inside thepytester.makepyfilestring of the test that tests it, so the guard rejected its own corpus. Walking real call nodes is the only version that distinguishes code from a string holding code.2. Three shapes that turn a whole session green
The other guards ask whether a test asserted anything. These six ask whether the run did.
parametrizeover an empty list1 skipped, exit 0Read the first row precisely, because bare pytest is not silent: it exits 1 and prints
worker 'gw1' crashed. What it never mentions istest_loss.py::test_d, which was collected, assigned to the dead worker, and never ran. A suite whose crash lands on a test already expected to fail reports exactly what you expected while running fewer tests than you wrote.Half of these six are controls, deliberately: a run-shape guard fires on the whole session, so a false positive costs the entire suite rather than one test.
Two things took a measurement to get right:
pytest_xdist_node_collection_finishedis the only place the controller learns what was found.pytester.runpytest()runs the inner session in-process, so module-level sets are shared between the layer's own tests and the sessions they drive. Measured before the fix: 44 tests passed and the run exited 1, because the outer session had inherited every inner run's collected ids and setup skips.3. Two refusals pinned to their own messages
test_ordered_rows_both_empty_names_its_own_refusaland its sibling. Both guards refuse a both-empty comparison, and so doesrows()underneath them — so an arm asserting only "the inner run failed" passes with any one of the three deleted. This is the subsumption case in its purest form.Verified
The totals in
TESTS.mdare recomputed from disk rather than taken from either side of the merge, andselftest/350checks them.🤖 Generated with Claude Code
https://claude.ai/code/session_01Uf6UoeBRZYLQZa4KxNiw8a